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MULTEMODE FEEDBACK DISPLAY TECHNOLOGY 

The present invention relates generally to a system for providing 
haptic information to the human body. Haptic or tactual perception, 
5 commonly referred to as the "sense of touch", is actually composed of two 
parts: the cutaneous sense and kinesthesis [Loomis & Lederman, 1986]. 
The cutaneous sense mediates awareness of stimulation at the skin, while 
kinesthesis provides information on body parts. Conscious experience of 
mechanical cutaneous stimulation is known as "tactile perception". 

10 

Background 

The sense of touch, of haptic sense, is complex. Unlike vision, touch 
involves a direct physical interaction, so that human actions cause and 
change what is perceived. It is the fundamental role of touch to sense the 

15 results of these contact interactions in order to guide manipulation and body 
motion. The current art only contains devices for presenting a crude tactile 
image to a location on the body. This type of information represents only a 
fraction of the total information sensed by the human sense of touch. Many 
different types of basic information about the contacts are extracted by 

20 people through a combination of four different tactile sensory systems, 

measurement of joint motion, and measurement of joint torques. This basic 
information can be organized broadly into passively acquired information 
and actively acquired information. Passively acquired information can be 
gathered without relative motion between the body and an object and without 

25 varying the forces between the body and the object. Actively acquired 
information requires either relative motion or varying contact forces. A 
summary of some of the basic types of cues or sensed information is given 
below. 

30 

1 
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Passively acquired identification cues 

Passively acquired identification cues can be obtained by the act of 
holding an object in the hand. The inventors herein have identified three 
major cues in this category. 

1 . Determining the local contact normal and curvature of the surface 
at each contact. 

Determining the local contact curvature and labeling it as a point, 
edge, or planar contact is important for grasp acquisition and object 
identification. Different contact types have different grasp properties 
which affect stability [Salisbury, 1985]. In addition, the local contact 
normal and curvature provide a strong pruning heuristic rule for 
identifying objects and object pose; Grimson, W.E.L., Object 
Recognition by Computer: The Role of Geometric Constraint, MIT 
Press, Cambridge, MA (1990). Local object curvature can be 
measured passively by examining the normal contact force 
distribution. 

2. Determining the surface texture at each contact. 

Surface texture affects grasp stability. Rough textures are generally 
easier to grasp than smooth textures. In addition, surface texture can 
also be used as a pruning heuristic rule in identifying objects and 
object pose. Texture cues are produced both from the spatial 
distribution of the contact force and the dynamic vibration effects 
produced at the skin during motion. 

3. Gross object shape. 

By using the finger geometry and the location of the contact points in 
the hand, the shape of the object can be estimated. For example, an 
initial grasp might be a power grasp in order to get a lot of contact 
information about an object. Once the object and its shape is 
identified, a dexterous grasp might be used for manipulation. 



WO 95/20787 



PCT/US95/01226 



Actively acquired identification cues 

With some local active exploration, properties relating the reaction 
forces to the applied force can also be determined: 

5 1. Determining the local stiffness at each contact. 

The local surface stiffness can be estimated by applying varying 
normal forces and measuring the change in contact area. Softer 
surfaces will produce a larger contact area for a given applied force. 
This is not a significant cue for manipulation of control surfaces and 
10 tools since almost all of the objects involved are rigid. 

2. Determining local frictional properties. 

The local friction along with the local contact type control the local 
grasp stability. Friction can be measured by applying varying 
tangential forces and then detecting the onset of slip (slip detection is 
15 a separate basic manipulation cue; see next Section). This task also 

requires the ability to measure tangential forces. 

3. Overall object stiffness. 

By integrating the contact force over an area with the joint torques, 
and correlating this with joint deflection, the overall object stiffness 
20 can be computed. 

4. Detecting slip. 

Detecting the onset of slip between an object and the hand is essential 
for grasp maintenance. Slip detection is used to determine the 
necessary grasp forces at each contact during all stages of 
25 manipulation. 

5. Determining object mass (weight), center of mass, and 
moments of inertia. 

By manipulating a grasped object and measuring the resulting joint 
torques and net contact forces at the hands for different 
30 configurations, the weight, center of mass and moments of inertia of 
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the object can be computed. This information can be used for object 
identification and pose determination, Siegal, D.M., Pose 
Determination of a Grasped Object Using Limited Sensing, Ph.D. 
Thesis, Department of Electrical Engineering, MIT, Cambridge, 
5 Massachusetts (1991), as well as for computing the necessary torques 

for throwing or manipulating the object. 

6. Estimating directions of contact constraint. 

Assembly is the process or bringing a moving part into a constrained 
relationship with a fixed part. In order to control the assembly 
10 process, the directions in which movement is constrained need to be 

estimated. Contact constraints are estimated using measurements of 
the reaction forces as a function of position and by measuring the 
direction of impact forces. 

7. Detecting changes in contact constraints. 

15 This is one of the most common tasks during manipulation. The 

detents in switches, the termination in screws, the impacts from 
mating two parts are all examples. The onset of the change can be 
detected by looking for unexplained impact forces and the direction of 
the impact force. 

20 

Review of Haptic Devices 
Position and Orientation Devices 

Devices and systems are well known for measuring or monitoring one 
or more characteristics of an anatomical part (see for example U.S. Patent 
25 Nos. 4,986,280, 3,258,007 and 3,364,929). A human controlled position 
sensing device for use in the field of robotics is also well known (see for 
example, U.S. Patent Nos. 4,328,621, 4,534,694, 4,608,525, and 
4,674,048). 

30 
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Force Devices 

Systems are also well known for providing force feedback to human 
operators. Force-reflective devices exist in a variety of forms, from 
joysticks to arm and hand controllers. These devices serve as input devices 

5 and haptic interfaces for virtual environment manipulation, or teleoperation. 
While joysticks are probably the oldest of the haptic interface 
technologies and were originally conceived to control aircraft, many 
joysticks currently developed are being used for the control of remote 
manipulators. AT&T Bell Laboratories has developed a high performance, 

10 compact force-feedback joystick It was intended to support engineering, 

business, and scientific desktop computing applications. It is 7x4.5x3 inches 
in size, weighs approx. 21bs, can apply 75 grams force to the end of the 
joystick (2 inches long), and can run at over 200 Hz. its use has been 
demonstrated for navigating through an electronics CAD database by setting 

15 the force to have a step jump at the edge of each trace. It is fast enough to 
allow the users to feel textures, and provides enough force to give a good 
range of levels for different conditions. 

A significant amount of research has been conducted on robot hands, 
Salisbury, J.K. and Mason, M.T., Robot Hands and the Mechanics of 

20 Manipulation, MIT Press, Cambridge, Massachusetts (1985), and force- 
reflecting controllers, (e.g. Bejezy and Salisbury, 1983, Agronin, 1987) for 
robot arm manipulation. These controllers can be anthropomorphic but are 
often tailored to the slave robots and imitate their kinematics. The 
advantage of controllers that imitate the kinematics of the human hands and 

25 arms is that they have the largest user range of motion. However, it is 
difficult to achieve high quality force feedback with these controllers and 
minimizing actuator size becomes a major design requirement. The 
Stanford/JPL 6 DOF hand controller is an example of a compact non- 
anthropomorphic force reflecting arm master. Sargent Industries/CRL 

30 manufactures a variety of master-slave manipulators for use in hazardous 

5 
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environments. Though their systems do provide force feedback, they are 
large, non-anthropomorphic, and are not meant to be portable. Similarly, 
Schilling Development, Inc. produces the TITAN 7F, and the GAMMA 7F, 
both of which are 6-4 Titanium constructed servo, hydraulic master/slave 

5 systems. The GAMMA 7F allows operation in radioactive environments. 
However, the force-feedback versions are still under development, and the 
master controllers are only semi-anthropomorphic. 

One complete anthropomorphic system (Jacobsen 1989) uses high 
performance hydraulic actuators to provide force-reflection for 7 DOF up to 

10 and including the wrist, with more DOF's in the gripper (Sarcos, 1991). 
The device has a large range of force exertion at a relatively high 
bandwidth. However, it is bulky and also very expensive. Two other 
anthropomorphic teleoperation systems include the Force-Reflecting Hand 
Controller (FRHC) (McAffee et al.), and the Odetics exoskeleton (Burke, 

15 1991). Both use steel cables to reflect forces and moments and remotely 

locate the motors. The FRHC provides motion in 6 DOF while the Odetics 
exoskeleton uses 7 DOF to represent the human arm. However, these 
systems are still not portable force-feedback devices. Iwata (Iwata 1990) 
describes a system for the thumb, two fingers and the palm. The palm is 

20 actuated by a six degree-of-freedom parallel stage driven by electric motors. 
Each of the two fingers and thumb has single DOF motion and is also driven 
by electric motors. This system can transmit large forces to the hand and 
fingers, but is limited to pinch type grasps because of the single DOF plates 
for the fingers. 

25 Burdea, Zhuang, and Roskos (1991) describe a system based on 

pneumatic cylinders and the VPL dataglove. This system generates forces 
using pneumatic cylinders placed between the inside of the palm and the 
fingertips. The system is compact and lightweight, but at present can only 
simulate grasp forces between the palm and the fingertips. Contact of the 

30 fingers with objects supported externally cannot be simulated. While many 
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forms of force-feedback devices have been developed, not many complete 
anthropomorphic systems exist. Of the anthropomorphic arm systems, the 
main disadvantage is their bulkiness and non-portability. 

5 Tactile Devices 

Tactile devices are also well known and fall into five basic categories. 
Tactile displays have been the subject of much research activity in the past 
thirty years. While neuroscientists studied the physiological roots of human 
tactile perception, engineers were interested in tactile display devices that 

10 substituted or augmented other senses such as sight and hearing. Tactile 

displays also demonstrated their usefulness in VR simulations and telerobotic 
operations by EXOS TouchMaster which utilized buzzers to display contact 
information of objects. To design a high performance force reflective 
exoskeleton master, tactile displays might provide the teleoperator with 

15 additional information such as contact locations that cold not be adequately 
displayed by force alone. Or tactile displays might simplify the requirement 
on force reflection. It was conceivable that a combination of tactile display 
and force reflection would be haptically more effective, technically less 
challenging and economically more feasible than force reflection alone. 

20 The following paragraphs present an overview of tactile display 

technologies and their applications developed in the past thirty years. 

Air or water jet displays 

Not all tactile displays make mechanical contact with the skin. Air or 

25 water jet displays are attractive because they eliminate the need for moving 
mechanical parts and because they exhibit a consistent frequency response. 
Air jet arrays were extensively tested by Bliss, J.C., Crane, H.D. and Link, 
S.W., Effect of Display Movement on Tactile Pattern Perception, Perception 
and Psychophysics, 1:195-202, (1966), and Bliss, J.C., Crane, H.D., 

30 Mansfield, P.K. and Townsend, J.T., Information Available in Brief Tactile 
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Presentations, Perception and Psychophysics, 1:273-283 (1966) in different 
configurations with applications in vision substitution Bliss, J.C., Crane, 
H.D. and Link, S.W., Effect of Display Movement on Tactile Pattern 
Perception, Perception and Psychophysics, 1:195-202, (1966), Rogers, 
5 C.H. , Choice of Stimulator Frequency for Tactile Arrays, IEEE Transactions 
on Man-Machine Systems (special issue), MMS-ll(l) (March, 1970) and as 
a user interface in tracking experiments, Hill, J.W., A Describing Function 
Analysis of Tracking Performance Using Two Tactile Displays, IEEE 
Transactions on Man-Machine Systems (special issue), MMS-ll(l) (March, 
10 1970). Steerable air and water jets were also investigated as a "substitute 
cathode ray tube", tracing cursive letters or figures or re-creating a visual 
image in a raster scanned format on the skin, Bliss, J.C., Tactual 
Perception: Experiments and Models, Stanford Research Institute, Menlo 
Park, California, Final Report, Technical Report AFAL-TR-66-242, 
15 Contracts AF33(615)-1099 and NAS2-2752, SRI Projects 4719 and 5438 
(July, 1966), and Collins, C.C. and Madey, J.M.J. , Tactile Sensory 
Replacement, Proceedings of the San Diego Biomedical Symposium, pp. 15- 
26 (1974). 

20 Air or water bladder displays 

Air or water bladder displays are used to apply a continuous pressure 
on an area of skin. An example is the TELETACT device (Air Muscle 
Ltd), which consists of an elastic lycra glove lined with approximately 
twenty air pockets fed by microcapillary tubes, Stone, R.J., Advanced 

25 Human-System Interfaces for Telerobotics Using Virtual Reality and 

Telepresence Technologies, Proceedings of the International Conference on 
Advanced Robotics , Pisa, Italy, 168-173 (June, 1991). This glove may be 
worn inside measurement devices such as the Mattel Power Glove or the 
EXOS Dextrous HandMaster (DHM) when interacting with a virtual 
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environment. Air pockets generally suffer from extremely low bandwidth 
(order of 1Hz) because of the elasticity of the bladder material. 

Mechanical tactor e lements or arrays that impact or indent the skin 

5 This group of displays include most of the vibrotactile devices 

described in the literature in addition to a few displays operated at a very 
low frequency that would not normally be considered vibrotactile (such as 
the tactile interface for IBM graphics described by Frisken-Gibson, S.F., 
Bach-y-Rita, P., Tompkins, W.J. & Webster, J.G., A 64 Solenoid, Four 

10 Level Fingertip Search Display for the Blind, IEEE Transactions on 
Biomedical Engineering, BME-34:963-965 (1987). 

While there are a few single-element display devices (see Goldstein, 
M.H. and Proctor, A., Tactile Aids for Profoundly Deaf Children, Journal 
of the Acoustical Society of America, 77:258-265 (1985), Terry, J.A. and 

15 Hsiao, H., Tactile Feedback in a Computer Mouse, Proceedings of the 
Fourteenth Annual Northeast Bioengineering Conference, p. 146-149 
(March, 1988)), the vast majority of mechanical tactile displays involve 
tactor arrays. The number of elements in these arrays run from two (such 
as the stump-mounted upper-limb prosthesis elbow angle display developed 

20 by Mann, R.W. and Reimers, S.D., Kinesthetic Sensing for the EMG 
Controlled "Boston Arm ", IEEE Transactions on Man-Machine Systems 
(special issue), MMS-11: 110-1 15 (1970) to 400 (such as the 20x20 back 
mounted array of solenoid activated tactors used by White, B.W., 
Perceptual Findings with the Vision-Substitution System, IEEE Transactions 

25 on Man-Machine Systems (special issue), MMS-11: 54-58 (1970), and 
Collins, C.C., Tactile Television - Mechanical and Electrical Image 
Projection, IEEE Transactions of Man-Machine Systems, MMS-11, 58-65 
(1970). 

Almost all of the devices in this group involve indenting or impacting 
30 the skin perpendicular to the skin surface. Two methods of actuation 

9 
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dominate these displays: solenoid activated pins and cantilevered 
piezoelectric bimorph reeds with a pin mounted to the tip. The OPTACON 
is an extremely well known piezoelectric bimorph activated vibrator array 
developed by Bliss et al (1970). It is a small 1.1x2. 7cm display with a 
5 matrix of 6x24 tactors, and can be operated from 12.5 Hz to 250Hz. It was 
originally designed to serve as a reading aid for the blind in conjunction with 
an optic probe which digitizes the information in printed text, but due to its 
versatile design it was used extensively as a research tool by a variety of 
researchers (such as Wiker, S.F., Vanderheiden, G., Lee, S. and Arndt, S., 

10 Development of Tactile Mice for Blind Access to Computers, Importance of 
Stimulation Locus, Object Size, and Vibrotactile Display Resolution, 
Proceedings of the Human Factors Society , 1: 708-712 (1991). The 
OPTACON is now commercially available from TeleSensory Systems. 
Another method of actuation that has great potential is the use of shape 

15 memory alloys (SMA). As a current is passed through the alloy, it changes 
shape and deflects a spring that holds a tactor pin in place; Johnson, A.D., 
Shape Memory Alloy Tactical Feedback Actuator, No. AAMRL-TR-90-039 
(available through NTIS), TiNi Alloy Co., Inc., Oakland, California, under 
sponsorship of Harry G. Armstrong Aerospace Medical Research 

20 Laboratory, Wright-Patterson AFB, Ohio (August, 1990), and McCarty, 
L.H., Special Alloy is Key to Braille Computer Display, Design News 
(Boston), 46(3): 158-159, (February 12, 1990). TACTOOLS is a 
commercial tactile feedback product available from Ztensory Inc. for use in 
teieoperation and virtual reality (VR) applications. The same product is also 

25 available from Mondo Tronics. 



Voice coil displays 

These displays generate vibratory sensations by mounting a voice coil 
against the skin. Examples are found in Patrick, N.J.M., Sheridan, T.B., 
30 Massimino, M.J., Marcus, B.M., Design and Testing of a Non-Reactive, 
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Fingertip Tactile Display for Interaction with Remote Environments, SPIE 
Vol. 1387 Cooperative Intelligent Robotics in Space, 215-222 (1990), and 
Patterson, P.E. and Katz, J.A., Design and Evaluation of a Sensory 
Feedback System that Provides Graspong Pressure in a Myoelectric Hand, 
5 Journal of Rehabilitation Research and Development, 29(1): 1-8, (Winter, 
1992). The TouchMaster is a voice-coil based tactile feedback device 
manufactured by EXOS. 

Electrotactile displays 
10 Electrotactile displays elicit touch sensations by indiscriminately 

stimulating all the touch and pain sensors in the skin with a current passing 
through the skin. These devices usually involve a remote stimulator (which 
supplies the power and generates the stimulation waveform) and a number of 
electrodes mounted on the skin using a sticky dielectric. Each electrode pair 
15 constitutes a stimulation site. While some systems involve single or a small 
number of isolated electrode pairs (such as the functional electrical 
stimulation feedback system developed by Rutherford, K.M., Riso, R., and 
Neuman, M.R., A Finger Mounted Force Sensor for Use with an FNS Hand 
Grasp System, Proceedings of the IEEE 9th Annual Conference of the 
20 Engineering in Medicine and Biology Society, pp. 1757-1758, arrays of 
electrodes were also popular, with number of elements running up to 1024 
in one instance, Collins, C.C. and Madey, J.M.J. , Tactile Sensory 
Replacement, Proceedings of the San Diego Biomedical Symposium, pp. 15- 
26 (1974). 

25 Large arrays with electrotactile displays are achieved by using small, 

closely spaced concentric electrode pairs in a grid, Collins, C.C. and 
Madey, J.M.J. , Tactile Sensory Replacement, Proceedings of the San Diego 
Biomedical Symposium, pp. 15-26 (1974). When the stimulation sites are 
few and sparsely distributed, a large common anode could be placed at a 

30 distance from the stimulation cites. The TICKLETALKER, Alcantara, J.I, 

11 
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Blarney, P.J., and Clark, G.M., Tactile-Auditory Speech Perception by 
Unimodally and Bimodally Trained Normal Hearing Subjects, Journal of the 
American Academy of Audiology, 4(2):98-108, (March, 1993) is one such 
device, in which 8 cathodes are mounted on four metal rings worn on each 
5 of the four fingers, and a large anode is mounted on the wrist. 

Current tactile feedback devices present a flat contact image using 
either arrays of tactors or raster scan techniques. This type of display can 
only be used for very local shape display or textures, and is limited to a flat 
type of display. Current force display devices can improve this situation 
10 when used in combination with tactile feedback, however the force displays 
have often geometrically limited the applicability of tactile displays. The 
concept of multimode feedback, that is the combination of one or more 
haptic modes in a single system, was developed specifically to overcome the 
limitations of the prior art. 

15 

Summary of the Invention 

It is therefore a general object of the current invention to provide a 
set of technologies and a method of providing touch or haptic information. 

It is a more specific object of the current invention to provide a set of 
20 technologies and a method which can be used to present critical information 
either separately or in combination. 

Another specific object of the current invention to provide a set of 
technologies and a method which can be used to present haptic cues either 
separately or in combination. 
25 And another specific object of the current invention is to provide a 

strategy for coordinating multiple feedback modes such as force and tactile. 

Yet another specific object of the present invention is to provide a 
haptic display which provides a continuous range of forces to the skin. 
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Yet another specific object of the present invention is to provide a 
haptic display which provides a time varying texture through relative 
tangential motion between the device and finger. 

These and other objects of the present invention are achieved by a 
5 system for displaying multimode feedback to an operator. The system 
comprises: 

A device for displaying tactile information to a body part; 
Means for tracking the position and orientation of the body part; 
Means for applying forces to the body part; 
10 Means for implementing a multidimensional control strategy; and 

Sensors for sensing body forces. 

Other objectives of the invention will in part be obvious and will in 
part appear hereinafter. The invention accordingly comprises an apparatus 
15 possessing the construction, combination of elements, and arrangements of 
parts which are exemplified in the following disclosure, and the scope of the 
application of which will be indicated in the claims. 

Brief Description of the Drawing 

20 Figure la and Figure lb are block diagrams of the preferred 

computerized system for obtaining and storing sensory information, 
controlling the displays, and communicating with other computers involved 
in the application; 

Figure 2 is an overview of the preferred haptic display unit; 
25 Figure 3 is a detail of the slip display element; 

Figure 4 is a concept sketch of an activated trackball based display; 
Figure 5 is a concept sketch of a two-degree-of-freedom roller 
display; 

Figure 6 is a concept sketch of an endless belt with various surface 
30 features mounted on a six degree-of-freedom mini-manipulator display; 

13 
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Figure 7 is a concept sketch of a display consisting of several actuated 
thimble rings with internal features; 

Figure 8 is a concept sketch of a rotating solid cylinder with different 
surface features at different angular positions; 
5 Figure 9 is a concept sketch of a display consisting of a rotating, flat 

slotted disk with an articulated linkage; 

Figure 10 is a concept sketch of a display with a piezoelectric 
bimorph actuator element mounted on top of a linear piezoelectric actuator 
element; 

10 Figure 11 is a concept sketch of a display in which steel pins are 

deflected by a varying magnetic field set up by an electromagnet array; 

Figure 12 is a block diagram and flow chart of the local processor as 
it interacts with the world modeling computer. 

The same numeral are used throughout the drawings to designate the 
15 same or similar parts. 

Description of the Preferred Embodiment 

The present invention generally relates to a system and method for 
providing touch feedback to the human operator and is adapted to have at 

20 least two applications. The system is useful in the control of computer 

generated or virtual environments and of real or robotic environments. (See 
Figures la and lb.) 

In the preferred embodiment as shown in Figures 2 and 3, four sense 
and display elements were used; slip on the finger tips, force feedback to the 

25 index finger and thumb, variable frequency and amplitude vibrotactile 

displays to the fingertips (the EXOS TouchMaster) and vertical resistance. In 
addition the device provides position and orientation of each of the system 
elements. The reference numerals are defined as follows: 
1 . Index finger 

30 2. Thumb 
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3. Tactile display (including slip and vibrotactile elements) 

4. Slip display motor 

5. Vibrotactile display 

6. Modified two-degree-of-freedom SAFiRE providing force feedback to 
5 the index finger 

7. Proximal linkage of the modified two-degree-of-freedom SAFiRE. 

8. Distal linkage of the modified two-degree-of-freedom SAFiRE. 

9. Cable conduits on the modified two-degree-of-freedom SAFiRE. 

10. Motor package for the index finger and thumb force reflection. 
10 mechanisms 

1 1 . Thumb force reflection device 

12. Strap mounted bracket for haptic feedback device 

13. Mounting bracket for haptic feedback device 

14. Proximal passive pivot 

15 15. Horizontally mounted linkage 

16. Distal passive pivot 

17. High resolution encoder 

18. Front linear way 

19. Front sliders 

20 20. Front slider mounting plate 

21. Second vertical cable 

22. Set of idler pulleys 

23. Motor actuating vertical degree-of-freedom through a cable system 
mounted on the rear slider 

25 24. Vertical support structure 

25. Heavy base 

26. Textured delrin surface 

27. Miniature bearing 

28. Set screw 

30 29. Mounting bracket for the slip and vibrotactile display elements 
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30. 


Finger strap 




31. 


Activated trackball 




32. 


Pinch rollers for driving the trackball 




33. 


Flat surface on trackball 


5 


34. 


Two-degree-of-freedom roller 




35. 


Endless belt 




36. 


Six-degree-of-freedom roller 




37. 


Surface features on the belt 




38. 


Actuated thimble 
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39. 


Independently actuated thimble rings 




40. 


Internal features inside thimble rings 




41. 


Indexing cylinder 




42. 


Surface features at different angular positions on the indexing cylinder 




43. 


Slotted flat disk 
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44. 


Actuated, articulated linkage 




45. 


Piezoelectric bimorph actuator element 




46. 


Linear piezoelectric actuator element 




47. 


Steel pins 




48. 


Array of electromagnets 


20 




In this embodiment a simple slip display 3 was developed. This slip 



display 3 consists of a delrin cylinder 26 with a textured surface, directly 
mounted on the shaft of a miniature gear motor 4. The cylinder 26 is 
supported on the other side with a miniature bearing 27. The motor is held 
by a set screw 28 in the mounting bracket. 

25 The motor 4 was powered by a linear motor amplifier (not shown) 

and controlled by an MC-series Motion Controller board (Motion 
Engineering Inc., CA.)(not shown). The major specifications for the slip 
display and the associated experimental setup are: the slip display must be 
able to provide endless slip; it must be able to overcome the friction 

30 imposed by the normal contact force (up to 2 lb) between the fingertip and 
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the force display; the maximum attainable speed of slip should be around 1 
in/sec; the weight of the slip display should be less than 0.5 lb. 

A voice coil vibrotactile display 5 which is comprised of a small 
magnet mounted in a shaped plastic housing. When excited by the control 
5 electronics it produces a variable frequency variable amplitude vibration to 
the housing. In the preferred embodiment this display 5 was attached to the 
mounting block of the slip display 3, and thus provides a vibratory stimulus 
to the fingertip. 

The slip display 3 was attached to the tip of each of the index finger 

10 force feedback apparatus 6 and the thumb force feedback apparatus 1 1 . 

Force reflection to the index finger 1 is provided by a modified two-degree- 
of-freedom SAFiRE (U.S.S.N. 07/961,259, filed October 15, 1992, entitled 
"Sensing and Force reflecting Exoskeleton") prototype, which consists of 
two cable-driven linkages, with the two motors 10 mounted on the forearm. 

15 The axes of rotation of the linkages are co-located with the PIP and MCP 
joints of the index finger. Force reflection to the thumb is provided by a 
one-degree-of-freedom device, also driven by cables from a remotized motor 
mounted on the forearm. 

The vertical resistance and position and orientation in space is 

20 provided by a boom. The vertical force reflection axis provided up to 2 lbs 
of virtual weight at the hand. It also supported the weight of the SAFiRE 
through counterbalancing. The stiffness of the vertical boom up to the 
SAFiRE attachment bracket exceeded 140 lb/in, the minimum stiffness 
required for a cantilevered beam to feel rigid to a human operator, Tan, 

25 B.T., Sensor Application to Space-Suit Glove, Tactile Sensors for Robotics 
and Medicine, J.G. Webster (Ed.), John Wiley and Sons, New York, 331- 
340 (1988). The resolution of the vertical and horizontal position sensors 
for the hand and index finger was 0.005 inch. Two linear ways were 
mounted on opposite sides of the vertical support structure, which was 

30 rigidly attached to a heavy base. 
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The drive mechanism for the vertical degree-of-freedom was located 
on the back of the support structure. The motor providing vertical force 
reflection was horizontally mounted on the slider riding in the rear linear 
way. A threaded capstan was directly mounted on the shaft of the motor 23. 
5 A 7x7, 0.032 inch diameter steel cable was wound 3-4 turns around the 
capstan. One end of the cable was terminated at the bottom of the vertical 
support structure. The other end was terminated on a cable tensioner at the 
top of the structure. As the motor rotated, the cable hoisted the slider up 
and down the rear linear way. 

10 The rear slider was connected through a second cable and a set of 

pulleys to a plate mounted on two sliders riding in a second linear way 18, 
mounted on the front side of the vertical support structure. Using two 
sliders instead of one greatly improved the twisting and bending load 
capacity of the front linear way. The two sliders were mounted on the plate 

15 through two pivots to minimize bending moments applied to the front linear 
way. 

Vertical forces were transmitted to the SAFiRE by a horizontally 
mounted linkage. The linkage consisted of a distal block attached to the 
front slider plate 20, proximal block, and a 12 inch long rod. The distal 

20 block housed the bearings and the encoder for the first passive pivot, also 
called the distal pivot. One end of the rod was pivoted about this block so 
that the linkage rotated freely in the horizontal plane. The other end of the 
rod was pivoted about a second bearing/encoder housing, also called the 
proximal pivot 14. The proximal mounting block was attached to the 

25 SAFiRE base with a rigid bracket. The length of this bracket was designed 
to position the hand so that the proximal pivot was between the tip of the 
index finger and the thumb. 

The motor for vertical force reflection was selected based on the 
power requirements. We would like the motor to produce a force of 2 lbs at 

30 the hand. We also estimated that the hand could move at a maximum 
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vertical speed of 8 in per sec. We selected 90W motor without a gearhead 
to provide a smooth operation with minimal friction. The HEHD is 
designed to produce a force of 2 lb. at the fingertip. The tactile display 
must produce a force that exceeds the corresponding frictional force, if the 

5 frictional coefficient between the finger and the display is estimated at 0.5, 
the tactile display must be able to produce 1 lb. of force. If the display 
moved past the finger at 1 in/s, the required power would be around 
0. 1 12W. A motor producing 0.25-0.5 W would therefore be a suitable 
actuator for this display. The MicroMo Series 1016 motor fits these 

10 requirements; it is 0.384 inch in diameter, 0.63 inch long and weighs 0.23 
oz. This is mounted in the space beyond the fingertip. 

This touch display hardware is interfaced to the world model, in this 
case implemented on a graphics rendering PC, through a local processor, in 
this case a VME card cage which houses a TIC30 DSP board, I/O boards, 

15 and a BiT3 VME-ISA bus adapter. 

The preferred embodiment is not the only viable implementation of 
the multimode feedback. Any one element, such as the slip display, when 
combined with position or orientation sensing becomes a multimode display 
and thus must use the strategy for coordination, in a simplified form. Each 

20 sense and display unit has the ability to provide haptic cues to the operator 
as do the following alternate embodiments. The alternate embodiments can 
be roughly classified into trackball ideas, roller and belt ideas, indexing 
ring/cylinder ideas, piezoelectrically actuated devices, magnetically actuated 
devices and other miscellaneous concepts. 

25 

Trackballs 

The trackball (Figure 4) concepts involve frictionally driven balls. 
The trackball can have two degrees-of-freedom and can produce lateral skin 
stretch as well as a sense of slip. However, it can not represent the nature 
30 of contact geometry. 

19 



WO 95/20787 



PCT/US9S/01226 



A variation on this theme involves cutting a flat surface on the ball. 
This would produce a plane as well as an edge on the ball depending on how 
it is oriented with respect to the fingertip. 



5 Roller and belts 

A single roller can represent endless slip in two directions. If the 
roller is rotated about the center of the fingertip, slip in any direction may 
be simulated. If it is pressed hard against the fingertip, lateral skin stretch 
may be induced (see Figure 5). A row of such rollers can represent slip as 

10 well as a rough simulation of a flat surface (as long as the rollers are small 
and closely spaced). 

An endless belt 35 (Figure 6) may provide a better representation of a 
flat surface. If the endless belt can be rotated on its side, the edge of the 
belt can represent an edge. If the endless belt is mounted on a 6 degree-of- 

15 freedom miniature manipulator 36, it can represent slip and lateral skin 

stretch in any direction. Surface features may be built into the belt so that 
different textures are represented by different sections of the belt. 



Indexing ring/cylinder 
20 A thimble 38 (Figure 7) made of several rings 39 with internal ridges 

may be built. By lining up the ridges on different rings, edges with 
different orientations may be created under the fingertip. Rotating each ring 
will also present a sense of slip in a direction normal to the longitudinal axis 
of the finger. 

25 Alternatively, a solid cylinder 41 (Figure 8) with different surface 

features at different angular positions may be built. By rotating the cylinder, 
different textures may be presented to the fingertip. 

A third variant involves a flat disk 43 (Figure 9) with a slot down the 
middle of the disk. A miniature articulated linkage 44 may be raised from 

30 the slot to create a corner. When the linkage is retracted, a flat circular 
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surface will be presented. Spinning the disk will create a sense of slip. 
Adding a rotational degree of freedom will allow the disk to be tilted with 
respect to the fingertip. 

5 Piezoelectrica) ly activated devices 

A piezoelectrically micromotor may be used to generate rotation in a 
small package. Alternatively, a deflecting piezoelectric actuator 45 may be 
mounted on top of a linear piezoelectric actuator 46 to produce a tiny 
manipulator that can push on the fingertip as well as drag it sideways for 

10 normal and lateral force representation (see Figure 10). 

Magnetically activated devices 

Using a complex magnetic field, ferrite dust may be made to form 
different shapes. Alternatively, by modulating the polarity and strength of 
15 an array of electromagnets 48, tiny steel pins 47 may be deflected to 
generate different shapes at the fingertip (see Figure 11). 

Others 

Other ideas include using a thermoelectric cooling chip mounted 
20 against the finger to act as a heat sink. Since most metals are cold to the 

touch, this artificially induced heat loss may generate an illusion of being in 
contact with metal. 

Certain changes may be made in the above apparatus without 
departing from the scope of the invention involved herein. It is intended 
25 that all matter contained in the above description or shown in the 

accompanying drawings shall be interpreted in an illustrative and not in a 
limiting sense. 
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CLAIMS 

1 . Apparatus for displaying tactile information to a body part 
comprising: 

5 means for tracking the position and orientation of the body part; 

means for applying forces to the body part; 
means for implementing a multidimensional control strategy; and 
sensors for sensing body forces. 

10 2. The apparatus of claim 1 wherein the means for implementing 

the control strategy comprises; 
a mounting bracket; 

a rotatable textured cylinder mounted in the bracket; and 
a motor adapted to rotate the cylinder according to the control 
15 strategy. 

3. The apparatus of claim 2 further including a second haptic 
display. 



22 



WO 95/20787 



PCT/US95/01226 



HUMAN 
OPERATOR 



HAPTIC 
DISPLAY 
HARDWARE 



WORLD 




LOCAL 
PROCESSOR 


MODELING 


COM. 


SYSTEM 







FIG. 1A 



LOCAL PROCESSOR 



WORLD 
MODEL 



HAPTIC 
COMPRESSION 



SENSOR 
PROCESSING 



SIMULATION 



CONTROL 
ALGORITHMS 



SENSORY 
INFORMATION 



HAPTIC DISPLAY 
COMMANDS 



HAPTIC 
DISPLAY 
HARDWARE 



FIG. IB 
1/7 

SUBSTITUTE SHEET (RULE 26) 



WO 95/20787 



PCT/US95/01226 




SUBSTITUTE SHEET (RULE 26) 



WO 95/20787 



PCT/US95/01226 




SUBSTITUTE SHEET (RULE 26) 



WO 95/20787 



PCT/US95/01226 



36 




SUBSTITUTE SHEET ("RULE 26) 



WO 95/20787 



PCT/US95/01226 




5/7 



SUBSTITUTE SHEET (RULE 26) 



WO 95/20787 



PCT/US9S/01226 



=^ W C£ 



W *r -5 
O LI 5 , , 

a: z 
< 



a. O 



5 z c/> 
O- £ 

z.zjz 

Z (/V 

< o o 

ID. O 



^ o 



s3 



in 
o a 
a: rz 

si 

O Z 



1A 
»- 

a 

" </> 

Offl< 

So 

§-8 

< 



<ScS* 



I 

CO 

o 

uj 5 — o & 



< 

fr- 
et: 

> 



in w 

o — < 

UJ Q. 

-> ^ to 
CD ^ 
O 



-J 00 
< K 

a 



ti- 
i/i 

MO 
o ^ 

I 

o 
to 



O 
Q. 



I ' 



< 



E 



CM 


12A 


» 

CM 


E 


FIG. 


fig. 



6/7 

SUBSTITUTE SHEET (RULE 26) 



WO 95/20787 



PCT/US9S/01226 




Q£ CO 

uj O 
o z 

05 

Z UJ 

uj q: 



15 

!2 




'1 

■ o 



REACTION 
ES 




FORCES 
TORQUES 


JOINT 
TORQUES 


T JOINT 
TO MOTOR 
ANDS 


CALCULATE 
FORC 


REACTION 
FORCES 




CONVERT 
TO JOINT 




! CONVER 
TORQUES 
COMM 




7/ 7 

SUBSTITUTE SHEET (RULE 2Q 



INTERNATIONAL SEARCH REPORT 



Inter. *nal Application No 

PCT/US 95/01226 



A. CLASSIFICATION OF SUBJECT MATTER 

IPC 6 G06F3/00 



According to International Patent Classification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 

IPC 6 G06F B25J 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practical, search terms used) 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


X 
A 


W0, A, 91 11775 (KRAMER JAMES F) 8 August 
1991 

see page 4, line 24 - page 5, line 34 
see page 6, line 12 - page 7, line 13 
see abstract 


l 

3 


X 
A 


IEEE INTERNATIONAL CONFERENCE ON ROBOTICS 

AND AUTOMATION, 

vol. 1, May 1993 GA,USA, 

pages 955-960, 

CALDWELL D. G. ET AL 'Enhanced Tactile 
Feedback (Tele-Tact ion) using a 
Multi-Functional Sensory System 1 
see page 955, left column, line 17 - page 
958, right column, line 33 


1 

• 3 ! 






/— 




| x [ Further documents are listed in the continuation of box C. 


|X | Patent family members are listed in annex- 


* Special categories of cited documents : -j- later document published after the international filing date 

or priority date and not in conflict with the application but 
"A" document defining the general state of the art which is not cited to understand the principle or theory underlying the 

considered to be of particular relevance invention 
"E* earlier document but published on or after the international »x* document of particular relevance; the claimed invention 

filing date cannot be considered novel or cannot be considered to 

* IT document which may throw doubts on priority daim(s) or involve an inventive step when the document is taken alone 

which is cited to establish the publication date of another 'y- document of particular relevance; the claimed invention 
citation or other special reason (as specified) cannot be considered to involve an inventive step when the 
•O* document referring to an oral disclosure, use, exhibition or document is combined w ^ one °^^°*"^ 

other means ments, such combination being obvious to a person skilled 

•P* document published prior to the international filing date but "** e ^ . , mmmtm MfPflt f 

later than the priority date claimed eV document member of the same patent family 


Date of the actual completion of the international search 


Date of mailing of the international search report 


19 May 1995 


2 a 05.35 




Name and mailing address of the ISA 

European Patent Office, P.B. 581 8 Patentlaan 2 
NL - 2280 HV Rijswijk 
Tel. ( + 31-70) 34f>2040, Tx. 31 651 epo nl, 
Fax: ( + 31-70) 340-3016 


Authorized officer 

Bailas, A 



Form PCT/ISA/310 (second theet) (July 1993) 



page 1 of 2 



INTERNATIONAL SEARCH REPORT 



Inter, jnal Application No 

PCT/US 95/01226 



C(Comimiation) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



IEEE VIRTUAL REALITY ANNUAL INTERNATIONAL 
SYMPOSIUM, 

18 September 1993 SEATTLE, WA, 
pages 256-262, XP 000457695 
HIROTA K. ET AL 'Development of Surface 
Display' 

see page 256, line 14 - page 257, line 16 
see page 258, line 1 - page 259, line 11 
see page 261, line 4 - line 21 



Form PCT/15 A/310 (CDntinuiUon of leoond thcet) (July 1992) 



page 2 of 2 



INTERNATIONAL SEARCH REPORT 

Information on patent family members 



Inter, .anal Application No 

PCT/US 95/01226 



Patent document 
cited in search report 



Publication 
date 



Patent family 
member(s) 



Publication 
date 



WO-A-9111775 



08-08-91 



US-A- 
AU-B- 
AU-B- 
AU-A- 
EP-A- 



5184319 
5752394 
649655 
7319991 
0513199 



02-02-93 
28-04-94 
02-06-94 
21-08-91 
19-11-92 



Form PCT/ISA/210 (patent fun Ely tnnex) (July 1993) 



